Apparatus and method for user input

ABSTRACT

A user input apparatus and method may measure, using a first sensor, surface input information that is applied to a surface of a user input apparatus, may measure, using a second sensor, orientation information that is input based on a physical quantity associated with a pose or a rotary motion of the user input apparatus, and may generate a content control signal, by combining the surface input information and the orientation information.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.13/895,649, filed May 16, 2013, which claims the priority benefit ofKorean Patent Application No. 10-2012-0070287, filed on Jun. 28, 2012,and Korean Patent Application No. 10-2012-0157834, filed on Dec. 31,2012, in the Korean Intellectual Property Office, the disclosures ofwhich are incorporated herein by reference.

BACKGROUND

1. Field

The following description relates to a user input apparatus and method.

2. Description of the Related Art

Compared to a method of controlling two-dimensional (2D) content, amethod of controlling three-dimensional (3D) content may require aninput of high degree-of-freedom (DOF). Thus, the 3D content controllingmethod may need to generate a content control signal for controlling the3D content by modifying an existing input channel using method or todiversify an input channel. For example, a method of adding a functionfor a 3D virtual space using an input device such as a keyboard, ajoystick, a button, and the like, and using the added function as aninput, a method of using, as an input, kinematic and action informationof a user on a real 3D space, or a method of measuring a distance when ahand of a user contacts on an input device and then moves, a forceapplied to the input device, or a displacement occurring due to theapplied force, and using a measured value as an input may be employed.

In the case of using kinematic and action information of a user on the3D space, an input method of projecting, onto a 2D screen, a motion ofthe user in a spatial mouse form or the 3D space may be employed. Theabove method may need to use another input channel for inputting threedirections. In the case of a mapping method on the 2D screen, a physicalspace to be mapped is constrained and thus, a motion of an input deviceand the user may be unsuitable for interaction.

SUMMARY

According to an aspect of one or more embodiments, user input apparatusmay include a first sensor to measure surface input information that isapplied to a surface of the user input apparatus; a second sensor tomeasure orientation information that is input based on a physicalquantity associated with a pose or a rotary motion of the user inputapparatus; and a signal processing unit to generate a content controlsignal by combining the surface input information and the orientationinformation.

The surface input information may include vertical information about acontact surface of the user input apparatus or horizontal informationabout the contact surface.

The orientation information may include a computed value from at leastone of a magnetic field, a tilt angle, an angular velocity, an azimuth,gravity, and an acceleration.

The signal processing unit may select a plane on a three-dimensional(3D) space of content from the orientation information, and may generatenew information in the 3D space by projecting the surface inputinformation onto the selected plane.

The signal processing unit may change an attribute associated with atranslation or a rotation of the content using the selected plane andthe new information in the 3D space.

The user input apparatus may further include a display unit to displaycontent based on the content control signal.

The user input apparatus may further include a feedback implementationunit to provide a feedback to a user to indicate a change in content bythe content control signal.

The feedback may include at least one of an audio feedback, a hapticfeedback, and a visual feedback.

In the case of implementing the haptic feedback, the feedbackimplementation unit may include at least one of a force transferringunit to transfer a force to the user input apparatus, a tactile displayunit to express a pressure distribution on the user input apparatus, atleast one vibrating unit to vibrate the user input apparatus, and astimulus transferring unit to provide the user with a stimulus bytactile feedback.

The stimulus transferring unit may transfer at least one of a stimulususing an electrostatic force, a cold temperature stimulus or a warmtemperature stimulus using a temperature difference, a stimulus using anair suction or exhaustion force, and a stimulus using an electrodecontact.

According to an aspect of one or more embodiments, a user inputapparatus may include a first sensor to measure force information thatis applied to an input surface of the user input apparatus; a secondsensor to measure orientation information that is input based on aphysical quantity of a pose or a rotary motion of the user inputapparatus; and a signal processing unit to generate a content controlsignal by combining the force information and the orientationinformation.

The user input apparatus may further include a detector to detect aposition of a point of action of force at which the force information isinput.

According to an aspect of one or more embodiments, a user inputapparatus may include a first sensor to measure contact informationabout a user contact on an input surface of the user input apparatus; asecond sensor to measure orientation information that is input based ona physical quantity associated with a pose or a rotary motion of theuser input apparatus; and a signal processing unit to generate a contentcontrol signal by combining the contact information and the orientationinformation.

According to an aspect of one or more embodiments, a user inputapparatus may include a first sensor to measure direction inputinformation; a second sensor to measure orientation information that isinput based on a physical quantity associated with a pose or a rotarymotion of the user input apparatus; and a signal processing unit togenerate a content control signal by combining the direction inputinformation and the orientation information.

According to an aspect of one or more embodiments, a user input methodmay include measuring, using a first sensor, surface input informationthat is applied to a surface of a user input apparatus; measuring, usinga second sensor, orientation information that is input based on aphysical quantity associated with a pose or a rotary motion of the userinput apparatus; and generating a content control signal by combiningthe surface input information and the orientation information.

According to an aspect of one or more embodiments, a user input methodmay include measuring, using a first sensor, force information that isapplied to an input surface of a user input apparatus; measuring, usinga second sensor, orientation information that is input based on aphysical quantity of a pose or a rotary motion of the user inputapparatus; and generating a content control signal by combining theforce information and the orientation information.

According to an aspect of one or more embodiments, a user input methodmay include measuring, using a first sensor, contact information about auser contact on an input surface of a user input apparatus; measuring,using a second sensor, orientation information that is input based on aphysical quantity associated with a pose or a rotary motion of the userinput apparatus; and generating a content control signal by combiningthe contact information and the orientation information.

According to an aspect of one or more embodiments, a user input methodmay include measuring, using a first sensor, direction inputinformation; measuring, using a second sensor, orientation informationthat is input based on a physical quantity associated with a pose or arotary motion of a user input apparatus; and generating a contentcontrol signal by combining the direction input information and theorientation information.

Additional aspects of embodiments will be set forth in part in thedescription which follows and, in part, will be apparent from thedescription, or may be learned by practice of the disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

These and/or other aspects will become apparent and more readilyappreciated from the following description of embodiments, taken inconjunction with the accompanying drawings of which:

FIG. 1 illustrates a configuration of a user input apparatus accordingto an embodiment;

FIG. 2 illustrates an example of a user input apparatus using a forceaccording to an embodiment;

FIG. 3 illustrates an example of a haptic feedback function applied to auser input apparatus according to an embodiment;

FIG. 4 illustrates an example of a user input apparatus using a contactmotion according to an embodiment;

FIG. 5 illustrates an example of a user input apparatus using adirection key according to an embodiment;

FIG. 6 illustrates an example of a user input apparatus using a ballaccording to an embodiment;

FIG. 7 illustrates a structure of a force sensor that is employed for afirst sensor according to an embodiment;

FIG. 8 illustrates a structure of a contact sensor that is employed fora first sensor according to an embodiment;

FIG. 9 illustrates a method of performing a content control functionusing a tangential force and an orientation according to an embodiment;

FIG. 10 illustrates a method of performing a content control functionusing a tangential movement of contact coordinates (position) and anorientation according to an embodiment;

FIG. 11 and FIG. 12 illustrate an example of generating athree-dimensional (3D) input vector using surface input information andorientation information according to an embodiment;

FIG. 13 illustrates a method of controlling an object or a cursor usingsurface input information and orientation information according to anembodiment;

FIG. 14 through FIG. 16 illustrate an example of moving a cursor usingsurface input information and orientation information according to anembodiment;

FIG. 17 and FIG. 18 illustrate an example of controlling rotation of anobject using surface input information and orientation informationaccording to an embodiment;

FIG. 19 illustrates a user input method according to an embodiment;

FIG. 20 illustrates a user input method according to an embodiment;

FIG. 21 illustrates a user input method according to an embodiment; and

FIG. 22 illustrates a user input method according to an embodiment.

DETAILED DESCRIPTION

Reference will now be made in detail to embodiments, examples of whichare illustrated in the accompanying drawings, wherein like referencenumerals refer to like elements throughout. Embodiments are describedbelow to explain the present disclosure by referring to the figures.

Terminologies used herein are defined to appropriately describe theembodiments and thus, may be changed depending on a user, the intent ofan operator, or a custom. Accordingly, the terminologies must be definedbased on the following overall description of the embodiments.

A user input apparatus according to an embodiment may receive twoinputs, such as a surface input applied by a user and an orientationinput of the user input apparatus, for example, and may generate acontent control signal capable of controlling content based on the abovetwo inputs.

FIG. 1 illustrates a configuration of a user input apparatus accordingto an embodiment.

Referring to FIG. 1, the user input apparatus may include a first sensor110 to measure surface input information that is applied to a surface ofthe user input apparatus, a second sensor 130 to measure orientationinformation that is input based on a physical quantity associated with apose or a rotary motion of the user input apparatus, and a signalprocessing unit 150 to generate a content control signal by combiningthe surface input information and the orientation information. However,the disclosure is not limited to the above. For example, the firstsensor and the second sensor may be integrated into a single sensor tomeasure both surface input information and orientation information.

The user input apparatus may measure the surface input information andthe orientation information, and may generate a content control signalby processing at least one of the measured surface input information andorientation information as input information. The user input apparatusmay control and display content or an object using the content controlsignal.

The surface input information may include at least one of verticalinformation about a contact surface of the user input apparatus andhorizontal information about a level of pushing applied to the contactsurface. The orientation information may include a variety ofinformation including at least one of a magnetic field, a tilt angle,angular velocity, an azimuth, gravity, and acceleration.

The user input apparatus may further include an input vector calculator120. The input vector calculator 120 may process the surface inputinformation in a form that may be used as input information.

Based on a sensor type and a degree of freedom (DOF), the second sensor130 may calculate a tilt angle with respect to the gravity, an angularvelocity, an azimuth with respect to the magnetic north, and a relativepose with respect to a predetermined reference, for example. The secondsensor 130 may express the orientation information using at least one ofEuler angles such as roll, pitch, and yaw, for example. The secondsensor 130 may also express the orientation information in a matrix formsuch as a pan, a tilt, a quaternion, a directional cosine matrix (DCM),for example. The second sensor 130 may also express the orientationinformation using a gesture measurement value of tapping, shaking, andthe like.

The user input apparatus may further include an orientation calculator140. The orientation calculator 140 may process the orientationinformation in a form that may be used as input information.

The signal processing unit 150 may select a plane on a three-dimensional(3D) space of content from the orientation information, and may generatenew information in the 3D space by projecting the surface inputinformation onto the selected plane.

The first sensor 110 and the second sensor 130 of the user inputapparatus may be located on a device that the user carries or uses. Thesignal processing unit 150 may be located within the user inputapparatus or may be configured as a separate unit. When the signalprocessing unit 150 is configured as the separate unit, the user inputapparatus may include a module for wireless communication (not shown)and a module for power supply (not shown) in order to processinformation.

The signal processing unit 150 may change an attribute associated with atranslation or a rotation of the content using the selected plane andthe new information in the 3D space. The signal processing unit 150 mayinclude an input generator 151 and an object controller 152.

The input generator 151 may generate a final input by inputting, asinput values, a direction of force and an apparatus orientation that arecalculated and received from the input vector calculator 120 and theorientation calculator 140, respectively. The object controller 152 maygenerate a content control signal for controlling the content based onthe generated final input.

The user input apparatus may further include a display unit 160 todisplay the content based on the content control signal. The displayunit 160 may be configured to be combined with the user input apparatus.The display unit 160 may be provided as a separate module with a size atwhich a user may readily use the content, and may also be provided as astand-alone device.

The user input apparatus may further include a feedback implementationunit 170 to provide a feedback to a user to indicate a change in thecontent by the content control signal. Using the feedback implementationunit 170, the user input apparatus may transfer a feedback to the userto be suitable for the changed content or target.

Depending on embodiments, the feedback implementation unit 170 mayprovide the user with a variety of feedbacks, such as an audio feedback,a haptic feedback, and a visual feedback, for example.

For example, even though not illustrated, in the case of implementingthe haptic feedback, the feedback implementation unit 170 may include atleast one of a force transferring unit, a tactile display unit, and atleast one vibrating unit, and a stimulus transferring unit. The forcetransferring unit may transfer a force to the user input apparatus, thetactile display unit may express a pressure distribution, and the atleast one vibrating unit may physically vibrate the user inputapparatus. The stimulus transferring unit may provide the user with astimulus by tactile feedback. The stimulus transferring unit maytransfer at least one of a stimulus using an electrostatic force, a coldtemperature stimulus or a warm temperature stimulus using a temperaturedifference, a stimulus using an air suction or exhaustion force, and astimulus using an electrode contact.

Depending on embodiments, the user input apparatus may include the firstsensor 110 to sense an input by a hand on a user contact portion, thesecond sensor 130 to sense orientation information associated with apose of the user input apparatus, and the signal processing unit 150 togenerate a content control signal by combining force information andorientation information.

The user input apparatus may measure, using the first sensor 110, forceinformation that is applied to an input surface of the user inputapparatus, and may measure, using the second sensor 130, orientationinformation that is input based on a physical quantity of a pose or arotary motion of the user input apparatus. Using the signal processingunit 150, the user input apparatus may generate a content control signalby combining the force information and the orientation information.

The user input apparatus may further include a detector (not shown) todetect a position of a point of action of force at which the forceinformation is input.

The signal processing unit 150 may select a plane on a 3D space ofcontent from the orientation information, and may generate newinformation in a 3D space by projecting the force information onto theselected plane. The signal processing unit 150 may change an attributeassociated with a translation or a rotation of the content using theselected plane and the 3D space information.

FIG. 2 illustrates an example of a user input apparatus using a forceaccording to an embodiment.

Referring to FIG. 2, a first sensor 210 of the user input apparatus is asensor capable of measuring force information and using the measuredforce information as input information. Any type of sensors of whichstates vary due to the force applied to the user input apparatus may beemployed as the first sensor 210.

For example, the first sensor 210 may include any sensor of whichphysical states, such as resistance, capacitance, voltage, magneticfield, optical wavelength, density, temperature, length, and volume, forexample, vary, and capable of measuring a value of a varying physicalstate.

Also, the first sensor 210 may include an elastic layer to easilytransfer a force around the first sensor 210, or may be a sensor made ofan elastic material. Also, the first sensor 210 may be coated with anelastic material, and thus, may have durability.

Depending on embodiments, the number of first sensors 210 andarrangement thereof may be diversified. For example, at least one firstsensor 210 may be consecutively or inconsecutively configured. Here, thefirst sensor 210 may include a base to support the first sensor 210 fromforce that is applied to a contact portion.

Hereinafter, a method of measuring force information will be describedbased on the assumption that a force sensor is used as a first sensor ofa user input apparatus according to an embodiment.

As described above, the user input apparatus may be variously embodiedbased on the number of force sensors corresponding to a first sensor andarrangement thereof, and a type of a second sensor and a DOF thereof.

A force sensor that is a first sensor may calculate information about apoint of action of force, a direction of force, magnitude of force, andduration of time in which the force is applied, for example, based on aconfiguration of the user input apparatus, the number of user inputapparatuses, and the arrangement thereof.

The force sensor may calculate the direction of force from an output ofthe force sensor and directly use a calculated value, or may classifythe direction of force as a predetermined direction based on thecalculated value. For example, the force sensor may express thedirection of force using a unit such as a degree or a radian, and mayclassify the direction of force into one of up/down/left/right, 4compass bearings, 8 compass bearings, and 12-hour expression, andthereby express the direction of force. The force sensor may express thedirection of force as a 3D direction based on force that is verticallyapplied.

The force sensor may calculate the magnitude of force from the output ofthe force sensor and directly use a calculated value, or may express themagnitude of force using a predetermined magnitude level. For example,the force sensor may express the magnitude of force using a physicalunit such as voltage, a Newton (N), and PSI, for example, and may alsoexpress the magnitude of force using a relative level such as high/low,strong/medium/weak, and levels 1 to 5, for example.

The force sensor may calculate, from the output of the force sensor,duration of time in which the force is maintained, and directly use acalculated value, or may express the duration of time as a predeterminedmagnitude level using the calculated value. For example, the forcesensor may express the duration of time using a physical unit indicatinga time such as a second, for example, and may also express the durationof time using a relative level such as short/long and levels 1 to 3, forexample. The force sensor may calculate the duration of time from theoutput of the force sensor and may define and use an input of force ofconsecutive sequence.

The user input apparatus may include a second sensor 220 to measureorientation information that is input based on a physical quantity of apose or a rotary motion of the user input apparatus. The second sensor220 may sense orientation information associated with the pose. Thesecond sensor 220 may measure the physical quantity associated with thepose or the rotary motion state of the user input apparatus using atleast one of an accelerometer, an angular velocity detector, a tiltsensor, a magnetic field sensor, an azimuth sensor, and a gravitysensor.

The user input apparatus may generate a content control signal bycombining the force information and the orientation information.

FIG. 3 illustrates an example of a haptic feedback function applied to auser input apparatus according to an embodiment.

The user input apparatus may generate a content control signal by using,as input information, force information and orientation information thatis measured using a first sensor 310 and a second sensor 320,respectively. The user input apparatus may control content based on thegenerated content control signal. The user input apparatus may include afeedback implementation unit to provide a user with a change in thecontent that occurs according to control of the content.

The user input apparatus may provide the user with at least one of anaudio feedback, a haptic feedback, and a visual feedback. As illustratedin FIG. 3, the user input apparatus may provide the user with a hapticfeedback 330. The user input apparatus may include a vibrating motor toprovide the user with the haptic feedback 330.

The user input apparatus may include a speaker to provide the user withan audio feedback, and may include a display module for providing visualinformation to provide a visual feedback.

Depending on embodiments, the user input apparatus may include afeedback implementation unit to provide at least one haptic feedback ofa stimulus using an electrostatic force, a cold temperature stimulus ora warm temperature stimulus using a temperature difference, a stimulususing an air suction or exhaustion force, and a stimulus using anelectrode contact.

As an example, the user input apparatus may include an electrostaticforce actuator, and may provide the user with a tactile feedback in aportion in which a force of the user input apparatus is input, using theelectrostatic force actuator. Using the electrostatic force actuatormounted to a portion in which force of the user input apparatus is inputor applied, the user input apparatus may provide the user with asensation of a button click or a sensation of friction at a moment whena force input occurs.

As another example, the user input apparatus may include a Peltier heatpump, and may provide the user with a tactile feedback about cold/warmtemperature using the Peltier heat pump. In this example, the user inputapparatus may provide the user with the cold/warm temperature accordingto content control of the user by including a hardware module such asthe Peltier heat pump, for example, but is not limited to the hardwaremodule such as the Peltier heat pump. Therefore, the user inputapparatus may include a variety of devices to provide the cold/warmtemperature.

The user input apparatus may include the first sensor 110 to measurecontact information about a user contact on an input surface of the userinput apparatus, the second sensor 130 to measure orientationinformation that is input based on a physical quantity associated with apose or a rotary motion of the user input apparatus, and the signalprocessing unit 150 to generate a content control signal by combiningthe contact information and the orientation information.

The contact information measured using the first sensor 110 may includeat least one of information about whether a user contact with the userinput apparatus, and changed contact coordinates (position).

The signal processing unit 150 may select a plane on a 3D space ofcontent from the orientation information measured using the secondsensor 130, and may generate new information in the 3D space byprojecting the measured contact information onto the selected plane. Thesignal processing unit 150 may change an attribute associated with atranslation or a rotation of the content using the selected plane andthe new information in the 3D space.

FIG. 4 illustrates an example of a user input apparatus using a contactmotion according to an embodiment.

Referring to FIG. 4, the user input apparatus may include a first sensor410 including a contact sensor to measure contact motion and to use themeasured contact motion as an input, and a second sensor 420 to measureorientation information that is input based on a physical quantityassociated with a pose or a rotary motion of the user input apparatus.

The first sensor 410 of the user input apparatus may include any type ofsensor capable of sensing a contact on the user input apparatus or achange in contact coordinates (position). For example, the first sensor410 may include any sensor capable of measuring a change value in aphysical state such as resistance, capacitance, voltage, opticalwavelength, and temperature, for example, and may include at least onesensor in which contact measurement elements are combined in an arrayform.

Hereinafter, a method of measuring contact information will be describedbased on the assumption that a contact sensor is used as a first sensorof a user input apparatus according to an embodiment.

A contact sensor that is a first sensor may be variously embodied basedon a touchable area, a resolution, and a sensor type. The contact sensormay calculate information about contact coordinates (position), amovement direction of contact coordinates, and whether a predeterminedgesture is input, for example. The contact sensor may also sense acontact level or a proximity level based on the sensor type.

For example, the contact sensor may calculate the movement direction ofcontact coordinates (position) from an output of the contact sensor andmay directly use the calculated movement direction of contactcoordinates, or may classify the movement direction of contactcoordinates as a predetermined direction based on a calculated value andthereby use the same as contact information.

The force sensor may express the movement direction of contactcoordinates using a unit such as a degree or a radian, and may calculatethe movement direction of contact coordinates that is expressed usingup/down/left/right, 4 compass bearings, 8 compass bearings, and 12-hourexpression, for example. Also, the contact sensor may calculate, ascontact information, a gesture with respect to a vertical direction,such as a light tap or a long tap, which is distinguishable fromhorizontal direction information.

As another example, the contact sensor may use, as contact information,a level of pushing based on a sensor type, and may calculate, from theoutput of the contact sensor, duration of time in which the contact ismaintained, and calculate the calculated duration of time as contactinformation using magnitude of input. For example, the contact sensormay express the duration of time in which the contact is maintainedusing a physical unit indicating a time such as second, for example, andmay calculate contact information that is expressed using a relativelevel such as short/long and levels 1 to 3.

The user input apparatus may generate a content control signal bycombining the contact information that is measured using the firstsensor 410 and the orientation information that is measured using thesecond sensor 420.

The user input apparatus may include the first sensor 110 to measuredirection input information, the second sensor 130 to measureorientation information that is input based on a physical quantityassociated with a pose or a rotary motion of the user input apparatus,and the signal processing unit 150 to generate a content control signalby combining the direction input information and the orientationinformation.

The first sensor 110 may include at least one unit key for inputting apredetermined direction, and may receive consecutive direction inputinformation by the at least one unit key.

The signal processing unit 150 may select a plane on a 3D space ofcontent from the orientation information, and may generate newinformation in the 3D space by projecting the direction inputinformation onto the selected plane.

FIG. 5 illustrates an example of a user input apparatus using adirection key according to an embodiment.

Referring to FIG. 5, the user input apparatus may include the directionkey as a first sensor 510. In this example, the first sensor 510 mayinclude any sensor capable of sensing a key manipulation of a directionkey corresponding to received input information when the inputinformation about a predetermined direction key, for example, a4-direction key, an 8-direction key, or a direction key of a consecutiveangle, is received.

Hereinafter, a method of measuring direction input information will bedescribed based on the assumption that a direction key is used as afirst sensor of a user input apparatus according to an embodiment.

A direction key of the first sensor 510 may be variously embodied basedon the number of keys or buttons, arrangement thereof, and an inputscheme thereof, for example. For example, the direction key may beconfigured so that the respective keys of various directions such asup/down/left/right may be distinguished and thereby be pushed todesignate an input of a predetermined single direction. Alternatively,the direction key may be configured so that a single key may receiveinputs of a plurality of directions and thus, may receive directioninformation based on a pushed position and a tilt angle of the body ofthe key in a state where the key is pushed. Also, in addition to aninput with respect to a horizontal direction through the direction key,the first sensor 510 may receive an input that is applied in a verticaldirection.

The user input apparatus may include the second sensor 520 to measureorientation information that is input based on a physical quantityassociated with a pose or a rotary motion of the user input apparatus.The user input apparatus may generate a content control signal bycombining the direction input information that is measured using thefirst sensor 510 and the orientation information that is measured usingthe second sensor 520.

FIG. 6 illustrates an example of a user input apparatus using a ballaccording to an embodiment.

Referring to FIG. 6, the user input apparatus may apply, as a firstsensor 610, a ball sensor that rolls, rotates, or revolves. The firstsensor 610 may include any sensor capable of detecting a level ofrotation by a rolling element. The first sensor 610 may measure arotation level by measuring a change rate of an elastic body thatconfines the rolling element or by counting the number of times that therolling element rotates.

The user input apparatus may include a second sensor 620 to measureorientation information that is input based on a physical quantityassociated with a pose or a rotary motion of the user input apparatus.The user input apparatus may generate a content control signal bycombining direction input information (a rotation level) that ismeasured using the first sensor 610 and the orientation information thatis measured using the second sensor 620.

FIG. 7 illustrates a structure of a force sensor that is employed for afirst sensor according to an embodiment.

Referring to FIG. 7, the first sensor may include the force sensor, andmay include a sensor 710 to measure force information of a user, acontact portion 720 on which the user contacts, and a supporter 730 tosupport the pressure transferred to the sensor 710.

FIG. 8 illustrates a structure of a contact sensor that is employed fora first sensor according to an embodiment.

Referring to FIG. 8, the first sensor may include a contact sensor 820,and may be configured to detect an input on a contact surface such asartificial skin or a touch screen, for example, and be fixed by asupporter 810.

FIG. 9 illustrates a method of performing a content control functionusing a tangential force and an orientation according to an embodiment.

Referring to FIG. 9, a user input apparatus according to an embodimentmay generate a content control signal using surface input informationand orientation information, and may control content using the generatedcontent control signal.

When an input mode is executed in operation 910, the user inputapparatus may monitor a sensor signal in operation 920. In operation930, the user input apparatus may determine whether a force is input.When it is determined that the force is input, the user input apparatusmay measure a magnitude of the input force and a tilt angle in operation940.

In operation 950, when it is determined that the force is input, theuser input apparatus may determine whether a tangential force of theinput force is within an effective range. When a magnitude of thetangential force is sufficiently large, for example, when a magnitude ofthe tangential force is within the effective range, the user inputapparatus may transform a force input value to a 3D spatial vector inputvalue through transformation based on the tilt angle. For example, theuser input apparatus may transform the force input value based on tiltangle information in operation 960, and may control content using thetransformed input in operation 970.

In operation 980, if it is determined that a tangential force of theinput force is within an effective range, the user input apparatus maydetermine whether a normal force is within the effective range. Whenmagnitude of the normal force is sufficiently large or an input isvalid, the user input apparatus may perform a content control functionbased on the normal input in operation 990.

FIG. 10 illustrates a method of performing a content control functionusing a tangential movement of contact coordinates and an orientationaccording to an embodiment.

Referring to FIG. 10, a user input apparatus according to an embodimentmay generate a content control signal using surface input informationand orientation information, and may control content using the generatedcontent control signal.

When an input mode is executed in operation 1010, the user inputapparatus may monitor a sensor signal in operation 1020. In operation1030, the user input apparatus may determine whether a touch is input.When the touch is determined to be input, the user input apparatus maymeasure a touch gesture and a tilt angle in operation 1040.

In operation 1050, the user input apparatus may determine whether atangential movement of contact coordinates is within an effective range.When the tangential movement is within the effective range, the userinput apparatus may transform a value of touch movement using theorientation information in operation 1060, and may control content usingthe transformed input in operation 1070.

When the tangential movement is not within the effective range, the userinput apparatus may determine that the input touch is a tap gesture. Inoperation 1080, the user input apparatus may determine whether a tapgesture recognition result is within the effective range. When the tapgesture recognition result is within the effective range, the user inputapparatus may perform a content control function based on the normalinput in operation 1090. Based on a predetermined input pattern, theuser input apparatus may use the surface input information as an inputvalue for a predetermined function such as select, cancel, or modeshift, for example.

The user input apparatus may generate an input using a 3D spatial vectorand may also generate an input value by selecting a plane in a 3Dvirtual space using the measured orientation information and byprojecting, onto the selected plane, a unit vector of input that isapplied to the surface of the user input apparatus based on the surfaceinput information.

For example, the user input apparatus may generate the input using the3D spatial vector by transforming a 2D tangential input vector through adirectional cosine transformation matrix including orientationinformation according to Equation 1:

$\begin{matrix}{\begin{bmatrix}V_{x} \\V_{y} \\V_{z}\end{bmatrix} = {{C(\theta)}\begin{bmatrix}v_{x} \\v_{y} \\0\end{bmatrix}}} & \lbrack {{Equation}\mspace{14mu} 1} \rbrack\end{matrix}$

In Equation 1, [v] denotes the 2D tangential input vector, C(θ) denotestransformation based on an Euler angle or quaternion, and [V] denotesthe 3D spatial vector.

FIG. 11 and FIG. 12 illustrate an example of generating a 3D inputvector using surface input information and orientation informationaccording to an embodiment.

Referring to FIG. 11, when a device is posed in an orientation parallelto a plane on which a normal vector coincides with a reference vector(noted as “ref”), the pose may be measured using a second sensor. Next,the measured 2D input vector within the body coordinate by a firstsensor may be projected onto the plane in a 3D space.

An indicator provided on the body may correspond to one of a directionof a force input f, movement of coordinates of a touch input, adirection input using a direction key, and a direction input using aroller or a ball.

Referring to FIG. 12, when a device is posed in an arbitrary orientationwith respect to a plane on which a normal vector coincides with areference vector (noted as “ref”), the pose may be measured using asecond sensor. Next, the measured 2D input vector within the bodycoordinate by a first sensor may be projected onto the plane in a 3Dspace.

A tangential input vector measured by body coordinates may be expressedon the corresponding surface, and magnitude of force that is applied tothe surface and measured in the horizontal direction may be mapped withvector magnitude on a virtual space on which 3D content is expressed.The above mapping relationship may be linear or nonlinear.

FIG. 13 illustrates a method of controlling an object or a cursor usingsurface input information and orientation information according to anembodiment.

Referring to FIG. 13, when a control mode is executed in operation 1310,a user input apparatus according to an embodiment may determine whetherthe control mode corresponds to a translation input mode or anorientation input mode in operations 1320 and 1340.

In the case of the translation input mode, the user input apparatus maymove an object and a cursor based on a magnitude and direction of theuser input in operation 1330. In the case of the orientation input mode,the user input apparatus may rotate the object and the cursor based onthe magnitude and direction of the user input in operation 1350.

Based on the control mode, a 3D input vector that is generated throughthe user input apparatus may be mapped with a physical quantityassociated with a horizontal translation on a virtual space in which 3Dcontent is expressed. For example, the physical quantity may include aposition, velocity, acceleration, and the like. Also, the generated 3Dinput vector may be mapped with a physical quantity associated with arotary motion. For example, the physical quantity may include an angularposition, angular velocity, an angular acceleration vector, and thelike.

FIG. 14 through FIG. 16 illustrate an example of moving a cursor usingsurface input information and orientation information according to anembodiment.

Referring to FIG. 14 through FIG. 16, a user input apparatus accordingto an embodiment may use a 3D input vector, generated as an input value,as a value used to move coordinates of a 3D target, a cursor, or acamera.

FIG. 17 and FIG. 18 illustrate an example of controlling rotation of anobject using surface input information and orientation informationaccording to an embodiment.

Referring to FIG. 17 and FIG. 18, a user input apparatus according to anembodiment may use a 3D input vector, generated as an input value, tochange a rotation state of the object.

The user input apparatus may measure a magnitude and duration of time ofan input from an input that is vertically applied to a user contactsurface, and may control content using the measured value. Also, whenthe measured magnitude of force exceeds a predetermined threshold, theuser input apparatus may perform the same functionality as a button, ormay change a control value based on the magnitude of force.

The user input apparatus may perform a predetermined function byrecognizing a duration of time in which the force is maintained or aconsecutive input pattern of the force. For example, when verticalpressure beyond a predetermined effective range is measured, the userinput apparatus may select an object at which a cursor is currentlylocated.

The user input apparatus may modulate magnitude of a velocity vector atwhich the object moves using the vertical pressure, and may express themodulation by changing a color, a size, and the like of the cursor.Also, when a short force is consecutively applied to the verticaldirection twice, the user input apparatus may shift a mode from anavigation mode to a system menu selection mode. On the contrary, theuser input apparatus may shift a mode from the system menu selectionmode to the navigation mode.

The user input apparatus may detect a direction, magnitude, and durationof time of force using a contact sensor and may also detect a point atwhich the force is acting.

The user input apparatus may generate a content control signal based onsurface input information and orientation information, and may provide afeedback in response to control of 3D content using the content controlsignal.

The user input apparatus may provide at least one of an audio feedback,a haptic feedback, and a visual feedback to the user.

For example, when the user input apparatus moves or rotates the objectin response to a user input, the user input apparatus may provide aneffect such as inertia, viscosity, elasticity, color, temperature,roughness, friction, warping, brokenness, fragileness, and bending, forexample, based on an attribute of the object or a virtual space in whichthe object is displayed. Also, the user input apparatus may configurethe aforementioned feedback implementation as a graphical element andmay also configure the above feedback effect as a predetermined tone orpitch sound. Also, the user input apparatus may express the feedbackeffect using force, distribution of pressure, vibration, flow ofvibration, and the like.

Also, the user input apparatus may provide a tactile feedback to theuser by including an electrostatic force actuator or a Peltier heatpump. For example, using the aforementioned module such as theelectrostatic force actuator, the user input apparatus may provide theuser with a tactile feedback in a portion in which a force of the userinput apparatus is input. Using the aforementioned module such as thePeltier heat pump, the user input apparatus may provide the user with atactile feedback about cold/warm temperature.

As an example, if a user controls an airplane object flying in a 3Dspace using the user input apparatus, when the airplane object makescontact with the sea in the 3D space according to control of theairplane object, the user input apparatus may provide the user with ahaptic feedback of a cold sensation occurring due to the above contact.When the airplane object lands at the runway, the user input apparatusmay provide the user with a sensation of contact by the landing such asa sensation of pressing a button, collision, and vibration, for example.

FIG. 19 illustrates a user input method according to an embodiment.

Referring to FIG. 19, in operation 1910, a user input apparatusaccording to an embodiment may measure, using a first sensor, surfaceinput information that is applied to the surface of the user inputapparatus. In operation 1920, the user input apparatus may measure,using a second sensor, orientation information that is input based on aphysical quantity associated with a pose or a rotary motion of the userinput apparatus.

In operation 1930, the user input apparatus may generate a contentcontrol signal by combining the measured surface input information andorientation information. In operation 1940, the user input apparatus mayprovide a feedback to a user to indicate a change in content by thegenerated content control signal.

FIG. 20 illustrates a user input method according to an embodiment.

Referring to FIG. 20, in operation 2010, a user input apparatusaccording to an embodiment may measure, using a first sensor, forceinformation that is applied to an input surface of the user inputapparatus. In operation 2020, the user input apparatus may measure,using a second sensor, orientation information that is input based on aphysical quantity of a pose or a rotary motion of the user inputapparatus.

In operation 2030, the user input apparatus may generate a contentcontrol signal by combining the measured force information andorientation information.

FIG. 21 illustrates a user input method according to an embodiment.

Referring to FIG. 21, in operation 2110, a user input apparatusaccording to an embodiment may measure, using a first sensor, contactinformation about a user contact on an input surface of the user inputapparatus. In operation 2120, the user input apparatus may measure,using a second sensor, orientation information that is input based on aphysical quantity associated with a pose or a rotary motion of the userinput apparatus.

In operation 2130, the user input apparatus may generate a contentcontrol signal by combining the measured contact information andorientation information.

FIG. 22 illustrates a user input method according to an embodiment.

Referring to FIG. 22, in operation 2210, a user input apparatus maymeasure direction input information using a first sensor. In operation2220, the user input apparatus may measure, using a second sensor,orientation information that is input based on a physical quantityassociated with a pose or a rotary motion of the user input apparatus.

In operation 2230, the user input apparatus may generate a contentcontrol signal by combining the measured direction input information andorientation information. For example, according to the aforementioneduser input method, the user input apparatus may measure a 2D input thatis applied by the user to the user input apparatus and a tilt angle ofthe user input apparatus, and may use the measured values as 3D inputinformation.

A user input apparatus according to embodiments may embody various userinputs through combination with an existing user input apparatus.

A user input apparatus according to embodiments may measure magnitudeand a direction of a surface input that is applied by the user to anapparatus and may also measure an orientation of the apparatus using anorientation sensor in the user input apparatus. Accordingly, withoutusing both hands or without using a supporting structure, the user inputapparatus may measure input information for controlling 3D content.

A user input apparatus according to embodiments may control contentdisplayed on a 3D display unit by employing a surface input applied toan apparatus and orientation information of the user input apparatus.Accordingly, the user input apparatus enables an intuitive andconvenient user input.

The above-described embodiments may be recorded in non-transitorycomputer-readable media including program instructions to implementvarious operations embodied by a computer. The media may also include,alone or in combination with the program instructions, data files, datastructures, and the like. The program instructions recorded on the mediamay be those specially designed and constructed for the purposes ofembodiments, or they may be of the kind well-known and available tothose having skill in the computer software arts. Examples ofnon-transitory computer-readable media include magnetic media such ashard disks, floppy disks, and magnetic tape; optical media such as CDROM disks and DVDs; magneto-optical media such as optical discs; andhardware devices that are specially configured to store and performprogram instructions, such as read-only memory (ROM), random accessmemory (RAM), flash memory, and the like. The computer-readable mediamay also be a distributed network, so that the program instructions arestored and executed in a distributed fashion. The program instructionsmay be executed by one or more processors. The computer-readable mediamay also be embodied in at least one application specific integratedcircuit (ASIC) or Field Programmable Gate Array (FPGA), which executes(processes like a processor) program instructions. Examples of programinstructions include both machine code, such as produced by a compiler,and files containing higher level code that may be executed by thecomputer using an interpreter. The described hardware devices may beconfigured to act as one or more software modules in order to performthe operations of the above-described embodiments, or vice versa.

Although embodiments have been shown and described, it would beappreciated by those skilled in the art that changes may be made inthese embodiments without departing from the principles and spirit ofthe disclosure, the scope of which is defined by the claims and theirequivalents.

What is claimed is:
 1. A user input apparatus, comprising: a firstsensor configured to measure force information applied to an inputsurface of the user input apparatus; a second sensor configured tomeasure orientation information that includes a physical quantity of apose, or a physical quantity of a rotary motion, of the user inputapparatus; and a signal processing unit configured to generate a contentcontrol signal, by combining the force information and the orientationinformation, wherein the signal processing unit is configured to:calculate a magnitude of a force, based on the measured forceinformation, calculate a tilt angle of the user input apparatus, basedon the measured orientation information, and generate a 3D inputinformation, by using a transform of the force information based on thetilt angle, when the magnitude of the force is greater than or equal toa predetermined value.
 2. The apparatus of claim 1, wherein the forceinformation comprises: vertical information in a direction normal to thesurface, horizontal information in a direction parallel to the surface,or a combination thereof.
 3. The apparatus of claim 1, wherein thesecond sensor comprises: an angular velocity detector, a tilt sensor, amagnetic field sensor, an azimuth sensor, a gravity sensor, or acombination thereof.
 4. The apparatus of claim 1, further comprising: adetector configured to detect a position of a point of action of force,at which the force information is input.
 5. The apparatus of claim 1,wherein the signal processing unit is further configured to: select aplane on a three-dimensional (3D) space of content from the orientationinformation; and generate new information in the 3D space by projectingthe force information onto the selected plane.
 6. The apparatus of claim5, wherein the signal processing unit is further configured to: changean attribute associated with a translation or a rotation of the content,using the selected plane and the new information in the 3D space.
 7. Auser input method, comprising: measuring, using a first sensor, forceinformation applied to an input surface of a user input apparatus;measuring, using a second sensor, orientation information that includesa physical quantity of a rotary motion of the user input apparatus; andgenerating a content control signal, by combining the force informationand the orientation information, wherein the generating comprises:calculating a magnitude of a force, based on the measured forceinformation, calculating a tilt angle of the user input apparatus, basedon the measured orientation information, and generating a 3D inputinformation, by using a transform of the force information based on thetilt angle, when the magnitude of the force is greater than or equal toa predetermined value.
 8. The method of claim 7, wherein the forceinformation comprises: vertical information in a direction normal to thesurface, horizontal information in a direction parallel to the surface,or a combination thereof.
 9. The method of claim 7, wherein the secondsensor comprises: an angular velocity detector, a tilt sensor, amagnetic field sensor, an azimuth sensor, a gravity sensor, or acombination thereof.
 10. A user input apparatus, comprising: a firstsensor configured to measure force information applied to an inputsurface of the user input apparatus; a second sensor configured tomeasure orientation information that includes a physical quantity of apose, or a physical quantity of a rotary motion, of the user inputapparatus; a signal processing unit configured to generate a contentcontrol signal, by combining the force information and the orientationinformation; and a feedback implementation unit configured to provide,in response to a change in content by the content control signal, ahaptic feedback of at least one of: a stimulus using an electrostaticforce, a cold temperature stimulus or a warm temperature stimulus usinga temperature difference, a stimulus using an air suction or exhaustionforce, and a stimulus using an electrode contact, wherein the signalprocessing unit is configured to: calculate a magnitude of a force,based on the measured force information, calculate a tilt angle of theuser input apparatus, based on the measured orientation information, andgenerate a 3D input information, by using a transform of the forceinformation based on the tilt angle, when the magnitude of the force isgreater than or equal to a predetermined value.
 11. A user inputapparatus, comprising: a first sensor configured to measure contactinformation, of a user contact on an input surface of the user inputapparatus; a second sensor configured to measure orientation informationthat includes a physical quantity of a pose, or a physical quantity of arotary motion, of the user input apparatus; and a signal processing unitconfigured to generate a content control signal, by combining thecontact information and the orientation information, wherein the signalprocessing unit is configured to: calculate a movement of contactcoordinates, based on the measured contact information, calculate a tiltangle of the user input apparatus, based on the measured orientationinformation, and generate a 3D input information, by using a transformof the contact information based on the tilt angle, when the movement ofcontact coordinates is greater than or equal to a predetermined value.12. A user input apparatus, comprising: a first sensor configured tomeasure contact information, of a user contact on an input surface ofthe user input apparatus; a second sensor configured to measureorientation information that includes a physical quantity of a pose, ora physical quantity of a rotary motion, of the user input apparatus; anda signal processing unit configured to generate a content controlsignal, by combining the contact information and the orientationinformation, wherein the signal processing unit is configured to:calculate a level of pushing, based on the measured contact information,calculate a tilt angle of the user input apparatus, based on themeasured orientation information, and generate a 3D input information,by using a transform of the contact information based on the tilt angle,when the level of pushing is greater than or equal to a predeterminedvalue.
 13. A user input apparatus, comprising: a first sensor configuredto measure contact information, of a user contact on an input surface ofthe user input apparatus; a second sensor configured to measureorientation information that includes a physical quantity of a pose, ora physical quantity of a rotary motion, of the user input apparatus; anda signal processing unit configured to generate a content controlsignal, by combining the contact information and the orientationinformation, wherein the signal processing unit is configured to:calculate duration of time in which the contact is maintained, based onthe measured contact information, calculate a tilt angle of the userinput apparatus, based on the measured orientation information, andgenerate a 3D input information, by using a transform of the contactinformation based on the tilt angle, when the duration of time isgreater than or equal to a predetermined value.
 14. A user input method,comprising: measuring, using a first sensor, contact information, of auser contact on an input surface of a user input apparatus; measuring,using a second sensor, orientation information that includes a physicalquantity of a rotary motion of the user input apparatus; and generatinga content control signal, by combining the contact information and theorientation information, wherein the generating comprises: calculating amovement of contact coordinates, based on the measured contactinformation, calculating a tilt angle of the user input apparatus, basedon the measured orientation information, and generating a 3D inputinformation, by using a transform of the contact information based onthe tilt angle, when the movement of contact coordinates is greater thanor equal to a predetermined value.
 15. A user input method, comprising:measuring, using a first sensor, contact information, of a user contacton an input surface of a user input apparatus; measuring, using a secondsensor, orientation information that includes a physical quantity of arotary motion of the user input apparatus; and generating a contentcontrol signal, by combining the contact information and the orientationinformation, wherein the generating comprises: calculating a level ofpushing, based on the measured contact information, calculating a tiltangle of the user input apparatus, based on the measured orientationinformation, and generating a 3D input information, by using a transformof the contact information based on the tilt angle, when the level ofpushing is greater than or equal to a predetermined value.
 16. A userinput method, comprising: measuring, using a first sensor, contactinformation, of a user contact on an input surface of a user inputapparatus; measuring, using a second sensor, orientation informationthat includes a physical quantity of a rotary motion of the user inputapparatus; and generating a content control signal, by combining thecontact information and the orientation information, wherein thegenerating comprises: calculating duration of time in which the contactis maintained, based on the measured contact information, calculating atilt angle of the user input apparatus, based on the measuredorientation information, and generating a 3D input information, by usinga transform of the contact information based on the tilt angle, when theduration of time is greater than or equal to a predetermined value. 17.A user input apparatus, comprising: a first sensor configured to measurecontact information, of a user contact on an input surface of the userinput apparatus; a second sensor configured to measure orientationinformation that includes a physical quantity of a pose, or a physicalquantity of a rotary motion, of the user input apparatus; a signalprocessing unit configured to generate a content control signal, bycombining the contact information and the orientation information; and afeedback implementation unit configured to provide, in response to achange in content by the content control signal, a haptic feedback of atleast one of: a stimulus using an electrostatic force, a coldtemperature stimulus or a warm temperature stimulus using a temperaturedifference, a stimulus using an air suction or exhaustion force, and astimulus using an electrode contact, wherein the signal processing unitis configured to: calculate a movement of contact coordinates, based onthe measured contact information, calculate a tilt angle of the userinput apparatus, based on the measured orientation information, andgenerate a 3D input information, by using a transform of the contactinformation based on the tilt angle, when the movement of contactcoordinates is greater than or equal to a predetermined value.
 18. Auser input apparatus, comprising: a first sensor configured to measurecontact information, of a user contact on an input surface of the userinput apparatus; a second sensor configured to measure orientationinformation that includes a physical quantity of a pose, or a physicalquantity of a rotary motion, of the user input apparatus; a signalprocessing unit configured to generate a content control signal, bycombining the contact information and the orientation information; and afeedback implementation unit configured to provide, in response to achange in content by the content control signal, a haptic feedback of atleast one of: a stimulus using an electrostatic force, a coldtemperature stimulus or a warm temperature stimulus using a temperaturedifference, a stimulus using an air suction or exhaustion force, and astimulus using an electrode contact, wherein the signal processing unitis configured to: calculate a level of pushing, based on the measuredcontact information, calculate a tilt angle of the user input apparatus,based on the measured orientation information, and generate a 3D inputinformation, by using a transform of the contact information based onthe tilt angle, when the level of pushing is greater than or equal to apredetermined value.
 19. A user input apparatus, comprising: a firstsensor configured to measure contact information, of a user contact onan input surface of the user input apparatus; a second sensor configuredto measure orientation information that includes a physical quantity ofa pose, or a physical quantity of a rotary motion, of the user inputapparatus; a signal processing unit configured to generate a contentcontrol signal, by combining the contact information and the orientationinformation; and a feedback implementation unit configured to provide,in response to a change in content by the content control signal, ahaptic feedback of at least one of: a stimulus using an electrostaticforce, a cold temperature stimulus or a warm temperature stimulus usinga temperature difference, a stimulus using an air suction or exhaustionforce, and a stimulus using an electrode contact, wherein the signalprocessing unit is configured to: calculate duration of time in whichthe contact is maintained, based on the measured contact information,calculate a tilt angle of the user input apparatus, based on themeasured orientation information, and generate a 3D input information,by using a transform of the contact information based on the tilt angle,when the duration of time is greater than or equal to a predeterminedvalue.